Repository navigation
feat(catalog): Catalog with an optional multi-index hashing (MIH) index - #65
Merged
Merged
Conversation
📊 Benchmark Comparison Results✅ All benchmarks within noise104 benchmarks compared, no significant changes detected. Legend: ✅ Faster (>5%) · |
hexhamming.Catalog(records, width, index=False) copies and validates a fixed-width catalog once and serves first/best/all queries, single and many, with the free functions' results, sentinels and tie-breaking. With index=True it builds an MIH index: m flat CSR bucket tables of s ~ log2(N)+2 bits (capped at 22), with u32 record ids. A query probes every key within t = max_dist // m bits of each query substring. By pigeonhole that finds every record within max_dist; each candidate is then verified with the byte kernel. Candidates are not deduplicated or sorted. Ties go to the lower record index directly, and only all() sorts its matches. This avoids a per-query N-bit seen-set, which glibc would also have to zero. For each query, a cost model picks the index or a linear scan: probes times per-probe cost (rising with index size) against records times scan cost. It was fitted to a sweep of every probe radius 0..=4 on 8/16/32-byte catalogs of 10k-1M records. It chooses the faster path in all eight measured near-crossover cases; mispredictions elsewhere can only favour the linear scan. Python calls borrow exact bytes queries and detach the GIL only when the estimated work is at least 1 us, so small catalogs cost the same per call as the free functions. Apple M4 Max, 1M records, best_within, one query: width r free fn indexed speedup (build: 24 ms / 91 ms) 8 B 0 13.9 us 85 ns 164x (exact match early-exits) 8 B 2 114 us 82 ns 1392x 8 B 8 114 us 1.29 us 88x 8 B 12 108 us 56.4 us 1.9x 32 B 8 178 us 144 ns 1237x 32 B 32 177 us 5.77 us 31x 64x16 B catalog, no index: 59.8 ns vs free function 58.9 ns Index memory is sum_j 4 * (2^s_j + 1 + N) bytes: 45 MB for 1M x 8 B and 182 MB for 1M x 32 B. Tests: differential tests against the free functions across widths, radii and many-query variants. A new test drives the indexed paths directly at every probe radius 0..=4, with planted ties and repeated candidates, so the index path is covered even where the planner would scan. Python tests cover the class API, validation and the docs examples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
mrecachinas
force-pushed
the
feat/catalog-mih
branch
from
September 23, 2026 11:59
2ea5643 to
985f538
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
hexhamming.Catalog(records, width, index=False). It copies and validates a fixed-width catalog once, then servesfirst/best/allqueries (single and many). Results, sentinels and tie-breaking match the free functions.index=Truebuilds a multi-index hashing (MIH) index:mflat CSR bucket tables, eachs ≈ log2(N)+2bits (capped at 22), withu32record ids.t = max_dist // mbits of each query substring. By pigeonhole this finds every record withinmax_dist.all()sorts its matches. This avoids a per-query N-bit seen-set.Planner. For each query, a cost model chooses between the index and a linear scan. It compares probes × per-probe cost (which rises with index size) against records × scan cost.
bytesqueries and detach the GIL only when the estimated work is ≥ 1 µs, so small catalogs cost the same per call as the free functions.Apple M4 Max, 1M records,
best_within, one query:Index build takes 24 ms (8 B) / 91 ms (32 B). Memory is
Σ 4·(2^s_j + 1 + N)bytes: 45 MB / 182 MB. A 64×16 B catalog without an index costs 59.8 ns per call vs 58.9 ns for the free function.Tests